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Abstract 


Active Learning Architecture 


The analysis of a safety-critical system often requires detailed knowledge of safe regions and their high- 
dimensional non-linear boundaries. We present a statistical approach to iteratively detect and char- 
acterize the boundaries, which are provided as parameterized shape candidates. Using methods from 
uncertainty quantification and active learning, we incrementally construct a statistical model from only 
few simulation runs and obtain statistically sound estimates of the shape parameters for safety bound- 
aries. 


Introduction 



• All spacecraft, aircraft, and other complex systems can only work safely within a given operational 
envelope (Figure shows the flight path (red) of the ill-fated flight AF447 as altitude over mach number; 
important boundaries are shown in gray colors) 

• Multiple, non-linear boundaries in a high-dimensional parameter space and slow/expensive simulation 
runs limit the use of current analysis techniques like single- variable and linear techniques. 

• We use statistical emulation and hierarchical Bayesian modeling to quantify the uncertainites in models 
and make reliable predictions of complex phenomena like number, location, and shapes of boundaries. 
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1 . Select candidate point which 
maximizes E(l(X,a,alpha)) 

2. Run simulation with candidate point 

and obtain result 

3. Update DT with the candidate point 



• General goal: candidate points should be near boundaries 

• Maximum entropy Y = — Ecgci c Pc log Pc is too greedy 

• Active Learning McKay (ALM) : select maximum variance 

• Active Learning Cohn (ALC) : maximize reduction in predictive variance 

• Expected Improvement (El): maximize posterior expectation of improvement statistic 
Limitation: ALM, ALC, El do not take boundaries into account. 


Our Extension: Boundary-EI 


Architecture Overview 
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• We use DynaTrees: dynamic regression trees and sequential tree model for online applications 
[Taddy,Gramacy, Poison 2011] 

— Recursive partition of input space 

— Particle learning for posterior simulation 


• Focus on x with 0.5 — e < y(x) < 0.5 + e for 0 < e 

• Improvement (Jones 1998, Ranjan 2008): I(x) = e 2 (x) — min{(y(a;) — 0.5) 2 , e 2 (x)} 

• Expectation of I{x)\ ( a > 0, e(x) = as(x), std deviation s(x), y(x ) ~ N(y(x), s 2 (x)) 
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• Term 1 variability of response in e neighborhood 

Term 2 farther away and in areas with high variance 
Term 3 is active close to estimated boundary 
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solved with resampling and propagation 

• High efficiency through tree-based partitioning in higher dimensions 

• Particle mechanism suitable for active learning and experimental design 
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Bounday-EI 


Modeling Boundary Shapes 


• Task: estimate shapes of boundaries given points X n near boundaries 

• Boundary shapes can incorporporate physics and domain knowledge 

• Shape dictionary can be provided by domain expert 



Experimental Result 


Uncertainty in TTSAFE (Terminal Tactical Separation Assured Flight Environment) track data. We 
analyzed TTSAFE behavior with respect to bias in the measured Radar data. 



Summary 


• We developed a statistical framework to support analysis and uncertainty quantification of non-linear 
complex systems. 

• We used Bayesian statistical methology in combination with active learning techniques for efficient 
detection and characterization safety regions and their boundaries. 

• Case studies include NASA Intelligent Flight Control System (IFCS) and Terminal Tactical Separation 
Assured Flight Environment (TTSAFE) for Next Generation Air Traffic Control. 

• Future work will focus on further uncertainty quantification study, optimization of the active learning 
for high dimensional spaces, and application of the framework to other domains. 
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